iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer

In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer. Despite their weak induced bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this paper we generalize the correspondence to the recently introduced hierarchical Hopfield network, and find iMixer, a novel generalization of MLP-Mixer model. Unlike ordinary feedforward neural networks, iMixer involves MLP layers that propagate forward from the output side to the input side. We characterize the module as an example of invertible, implicit, and iterative mixing module. We evaluate the model performance with various datasets on image classification tasks, and find that iMixer reasonably achieves the improvement compared to the baseline vanilla MLP-Mixer. The results imply that the correspondence between the Hopfield networks and the Mixer models serves as a principle for understanding a broader class of Transformer-like architecture designs.Comment: 12 pages, 3 figure

Attention in a family of Boltzmann machines emerging from modern Hopfield networks

Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for a special case and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the Gaussian--Bernoulli restricted BM and denoising autoencoder with softmax units. We also investigate BMs introduced by other energy functions, and in particular, observe that the energy function of dense associative memory models gives BMs belonging to Exponential Family Harmoniums.Comment: 12 pages, 1 figur

Effect of acute ether or restraint stress on plasma corticotropin-releasing hormone, vasopressin and oxytocin levels in the rat.

Ether and restraint stress-induced peripheral plasma corticotropin releasing hormone (CRH), arginine vasopressin (AVP), oxytocin (OXY) and adrenocorticotropin (ACTH) levels were measured by radioimmunoassays. Plasma CRH, AVP, OXY and ACTH rose to approximately twice the level of control rats 2 min after the onset of a 1-min exposure to ether. Plasma CRH rose further 5 min after the onset of ether stress, while plasma AVP and OXY returned to the baseline levels at 5 min. Plasma CRH, OXY and ACTH showed significant elevation 2 min after the onset of restraint stress, while plasma AVP did not show a significant change. Plasma OXY and ACTH rose further 5 min after the onset of restraint stress, whereas plasma CRH returned to baseline levels. CRH and OXY concentrations in the hypothalamic median eminence decreased 5 min after the onset of ether exposure and restraint, while the AVP concentration did not differ from control levels. The results, including the discrepancy between plasma CRH and ACTH 5 min after stress, suggest that CRH in the peripheral plasma is derived from both hypothalamic and extrahypothalamic tissues. The levels of stress-induced CRH in the peripheral plasma were sufficient to stimulate ACTH release. These results suggest that ether and restraint stress elevate plasma CRH shortly after the onset of the stress, and that this elevation in the plasma CRH level is at least partly responsible for stress-induced ACTH secretion.</p

HLA-Haploidentical Peripheral Blood Stem Cell Transplantation with Post-Transplant Cyclophosphamide after Busulfan-Containing Reduced-Intensity Conditioning

AbstractAllogeneic hematopoietic stem cell transplantation (allo-SCT) using post-transplant cyclophosphamide (PTCy) is increasingly performed. We conducted a multicenter phase II study to evaluate the safety and efficacy of PTCy-based HLA-haploidentical peripheral blood stem cell transplantation (PTCy-haploPBSCT) after busulfan-containing reduced-intensity conditioning. Thirty-one patients were enrolled; 61% patients were not in remission and 42% patients had a history of prior allo-SCT. Neutrophil engraftment was achieved in 87% patients with a median of 19 days. The cumulative incidence of grades II to IV and III to IV acute graft-versus-host disease (GVHD) and chronic GVHD at 1 year were 23%, 3%, and 15%, respectively. No patients developed severe chronic GVHD. Day 100 nonrelapse mortality (NRM) rate was 19.4%. Overall survival, relapse, and disease-free survival rates were 45%, 45%, and 34%, respectively, at 1 year. Subgroup analysis showed that patients who had a history of prior allo-SCT had lower engraftment, higher NRM, and lower overall survival than those not receiving a prior allo-SCT. Our results suggest that PTCy-haploPBSCT after busulfan-containing reduced-intensity conditioning achieved low incidences of acute and chronic GVHD and NRM and stable donor engraftment and low NRM, particularly in patients without a history of prior allo-SCT

Effect of Hyperosmotic Stimulation and Adrenalectomy on Vasopressin mRNA Levels in the Paraventricular and Supraoptic Nuclei of the Hypothalamus:

The effects of salt loading and adrenalectomy on arginine vasopressin (AVP) mRNA levels in the paraventricular nucleus (PVN) and the supraoptic nucleus (SON) of the hypothalamus were studied by semiquantitative in situ hybridization histochemistry, using a synthetic oligonucleotide probe and a computer-assisted image analysis system. Salt loading (2% NaCl) for 7 days produced marked increases in AVP mRNA levels in the magnocellular neurons of the PVN, SON, and accessory nuclei. Adrenalectomy caused an increase in AVP mRNA expression in the magnocellular part of the PVN and the expansion of hybridization signals into its medial parvocellular region, where the cell bodies of corticotropin-releasing hormone (CRH) neurons are located. No apparent alteration of AVP mRNA levels was observed in the SON following adrenalectomy. These results indicate that hyperosmotic stimulation and the loss of circulating glucocorticoids had differential effects on AVP gene expression in the PVN and SON, and that the magnocellular PVN and SON neurons responded in different manners to the loss of feedback signals.</p

Combined anterior pituitary function test using CRH, GRH, LH-RH, TRH and vasopressin in patients with non-functioning pituitary tumors.

We examined 8 normal subjects and 16 patients with non-functioning pituitary tumors with a combined anterior pituitary test to evaluate the clinical usefulness of the test. Diagnoses included 9 of chromophobe adenoma, 3 of craniopharyngioma, 2 of Rathke's cleft cyst, and 1 each of intrasellar cyst and tuberculum sella meningioma. All subjects received hypothalamic releasing hormones: 1 micrograms/kg corticotropin releasing hormone (CRH), 1 micrograms/kg growth hormone releasing hormone (GRH), 500 micrograms thyrotropin-releasing hormone (TRH), 100 micrograms luteinizing hormone releasing hormone (LH-RH), and a relatively small dose (5 mU/kg) of lysine vasopressin (LVP). In the normal subjects, the addition of LVP potentiated the secretion of adenocorticotropic hormone (ACTH) induced by CRH, but had no significant effect on the secretion of other anterior pituitary hormones. In the combined test with 5 releasing hormones, the plasma ACTH and cortisol responses were not impaired in the majority of the patients before pituitary surgery. Serum thyroid-stimulating hormone (TSH), prolactin (PRL) and follicle-stimulating hormone (FSH) responses were not impaired in 82%, 70% and 67% of the patients, respectively, while the serum LH and GH responses were impaired in 67% and 73% of the patients, respectively. Following pituitary surgery, responses of these hormones to combined testing were similarly impaired in more than 75% of the patients. These results indicate that plasma ACTH, cortisol and serum TSH responses are fairly good before pituitary surgery but are impaired significantly after surgery. No subjects experienced any serious adverse effects related to the testing. These results suggest that combined testing with hypothalamic hormones is a convenient and useful method for evaluating pituitary function.</p